Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

/ok to test

PR 2140: First CI analysis of 13.3.0 failures

Workflow run inspected:

• https://github.com/NVIDIA/cuda-python/actions/runs/26521353069

Summary

The failures are not affecting all CUDA 13.3.0 test lanes. The CUDA 13.3.0
wheel-based test lanes are passing. The failures are concentrated in CUDA
13.3.0 LOCAL_CTK: '1' lanes, i.e. lanes that assemble and use a local mini
CUDA Toolkit via .github/actions/fetch_ctk.

The common failure mode is NVRTC compilation failing because CUDA C++/CCCL
headers are missing from the local mini-CTK:

NVRTC_ERROR_COMPILATION
catastrophic error: cannot open source file "cuda/atomic"

Other failing tests show the same class of missing-header problem:

catastrophic error: cannot open source file "cuda/std/complex"

and on Windows example tests:

cuda_toolkit\include\cooperative_groups/details/info.h(50): catastrophic error:
cannot open source file "nv/target"

Observed CI Pattern

Passing:

• Build jobs for CUDA 13.3.0 pass across Linux, Linux aarch64, and Windows.
• CUDA 13.3.0 (wheels) test lanes pass.

Failing:

• CUDA 13.3.0 (local) test lanes fail across Linux, Linux aarch64, and
  Windows.

Representative failed jobs include:

• Test linux-64 / Python 3.11, CUDA 13.3.0 (local), GPU l4
• Test linux-aarch64 / Python 3.11, CUDA 13.3.0 (local), GPU l4
• Test win-64 / Python 3.10, CUDA 13.3.0 (local), GPU rtxpro6000 (TCC)

Root Cause

The fetch_ctk action still requests the CCCL redistrib component under the old
component name:

cuda_cccl

For CUDA 13.3.0, the redistrib metadata no longer uses that key. It uses:

cccl

Evidence from redistrib metadata:

redistrib_13.2.1.json:
cuda_cccl/linux-x86_64/cuda_cccl-linux-x86_64-13.2.75-archive.tar.xz

redistrib_13.3.0.json:
cccl/linux-x86_64/cccl-linux-x86_64-13.3.3.3.1-archive.tar.xz

The 13.3.0 logs show the helper filtering out the old component name:

Skipping unsupported CTK component 'cuda_cccl' for host-platform 'linux-64'
Skipping unsupported CTK component 'cuda_cccl' for host-platform 'win-64'

After that, the local mini-CTK is assembled without CCCL headers, so tests that
compile CUDA C++ code through NVRTC fail when they include headers such as:

• cuda/atomic
• cuda/std/complex
• nv/target

Why Wheel Lanes Pass

The wheel lanes do not rely on this locally assembled mini-CTK in the same way.
They install CUDA runtime/compiler-related Python wheels, so they do not hit the
missing cccl component in .github/actions/fetch_ctk.

This is why the apparent failure is broad across local 13.3.0 tests, but not
across all 13.3.0 tests.

Suggested Fix

Update the mini-CTK component resolution to handle the redistrib component
rename from cuda_cccl to cccl.

Reasonable approaches:

1. Teach ci/tools/fetch_ctk_redistrib.py to alias cuda_cccl to cccl when
   the metadata contains cccl but not cuda_cccl.
2. Or include both cuda_cccl,cccl in the default component list and rely on
   metadata filtering to keep only the one that exists.

The first option is cleaner because existing workflow inputs can keep using the
logical old name while the helper adapts to the redistrib metadata shape.

After the fix, rerun a small representative subset first:

• one Linux CUDA 13.3.0 (local) lane
• one Windows CUDA 13.3.0 (local) lane

If those pass, rerun the full PR CI.

/ok to test